Criteria For Evaluating Usability Evaluation Methods

Among interactive system developers and users there is now much agreement that usability is an essential quality of software systems. The introduction of usability evaluation methods (UEMs) to assess and improve usability in such systems has led to a variety of alternative approaches and a general lack of understanding of the capabilities and limitations of each. This confusion has intensified the need for practitioners and others to determine which methods are more effective, and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. This paper presents a practical discussion of factors, comparison criteria, and UEM performance measures that are interesting and useful in studies comparing UEMs. We have attempted to highlight major considerations and concepts, offering some operational definitions and exposing the hazards of some approaches proposed or reported in the literature. In demonstrating the importance of developing appropriate UEM evaluation criteria, we present some different possible measures of effectiveness, select and review studies that use two of the more popular measures, and consider the trade-offs among different criterion definitions. In sum, this work highlights some of the specific challenges that researchers and practitioners face when comparing UEMs and provides a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison.